Dealing with randomisation bias in a social experiment: The case of ERA

The UK Employment Retention and Advancement (ERA) programme has been evaluated by a large-scale randomised experiment. It has however emerged that due to the experimental set-up over one quarter of the eligible population was not represented in the experiment: some eligibles actively refused to be randomly assigned, while some were somehow not even offered the possibility to participate in random assignment and hence in ERA. The fact that ERA was a study and involved random assignment has significantly altered how the intake as a whole was handled, as well as the nature of the adviser/individual interaction in a way that would not have been the case had ERA been normal policy. The pool of participants has been both reduced and altered, which is likely to have led to some randomisation bias or, alternatively, to some loss in external validity in the experimental estimate for the effect on the eligible population. The beauty of the ERA set-up and data is that it offers the rare chance to formally measure the extent of randomisation bias or the loss in external validity. Specifically, the key objective of the paper is to quantify the impact that the full ERA eligible population would have been likely to experience had they been offered the chance to participate in ERA, and to assess how this impact for the full eligible group relates to the experimental impact estimated on the potentially self-selected and advisor-selected subgroup of study participants. We separately consider how to deal with non-participation when follow-up information on the outcomes of the non-participants is available (administrative data) or not available (survey data such as earnings). Non-response to the survey and/or to the earnings question among survey respondents can create additional issues when trying to recover the earnings effect of ERA for the full eligible population. JEL codes: C21, J18, J38 Acknowledgments: I would like to acknowledge the many helpful suggestions from Costas Meghir, as well as Electra Small for her hard data work and Ingun Borg, Mike Daly, Christine Daniels, Richard Dorsett, Lesley Hoggart, Phil Robins, Jim Riccio and Stacy Sharman for detailed comments on previous drafts. The usual disclaimer applies. Address for correspondence: Institute for Fiscal Studies, 7 Ridgmount Street, London WC1E 7AE UK. E-mail: barbara [email protected].